Soft Margins for Adaboost Produced as Part of the Esprit Working Group in Neural and Computational Learning Ii, Neurocolt2 27150

Recently ensemble methods like AdaBoost were successfully applied to character recognition tasks, seemingly defying the problems of overrtting. This paper shows that although AdaBoost rarely overrts in the low noise regime it clearly does so for higher noise levels. Central for understanding this fact is the margin distribution and we nd that AdaBoost achieves { doing gradient descent in an error function with respect to the margin { asymptotically a hard margin distribution, i.e. the algorithm concentrates its resources on a few hard-to-learn patterns (here an interesting overlap emerge to Support Vectors). This is clearly a sub-optimal strategy in the noisy case, and regularization, i.e. a mistrust in the data, must be introduced in the algorithm to alleviate the distortions that a diicult pattern (e.g. outliers) can cause to the margin distribution. We propose several regularization methods and generalizations of the original AdaBoost algorithm to achieve a soft margin { a concept known from Support Vector learning. In particular we suggest (1) regularized AdaBoost Reg using the soft margin directly in a modiied loss function and (2) regular-ized linear and quadratic programming (LP/QP-) AdaBoost, where the soft margin is attained by introducing slack variables. Extensive simulations demonstrate that the proposed regularized Ada-Boost-type algorithms are useful and competitive for noisy data.